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AN INTERIM REPORT ON PENNSYLVANIA'S FRAMEWORK 
FOR LEADERSHIP: FREQUENTLY ASKED QUESTIONS 


A new study analyzed data from the 2012/13 pilot of the Framework for Leadership (FFL), Pennsylvania's tool for evaluating the leadership 
practices of principals and assistant principals. Based on the FFL scores of 336 principals and 69 assistant principals evaluated by their 
supervisors in the 2012/13 school year, this interim report found that school leaders' scores from different categories of practices produced 
consistent assessments of their effectiveness. However, most scores were high, regardless of whether the school leaders made large or 
small estimated contributions to student achievement growth. These findings suggest that the FFL is based on a coherent definition of 
leadership guality, but more evidence is needed on whether this tool can distinguish effective school leaders from ineffective ones. The 
full report, Measuring school leaders' effectiveness: An interim report from a multiyear pilot of Pennsylvania's Framework for Leadership, is 
available at http://ies.ed.gov/ncee/edlabs/projects/project.asp/projectl D=343. 


PURPOSE OF THE STUDY 


1. Why was this study conducted? 

Legislation enacted in Pennsylvania (Act 82 of 2012) created a new system for 
evaluating principals and assistant principals. Starting in school year 2014/15, 
half of a school leader's annual evaluation must be based on a supervisor's 
assessment of the school leader's leadership practices. (The other half must be 
based on measures of student achievement.) 

As mandated by the legislation, the Pennsylvania Department of Education 
created the Framework for Leadership, a tool for supervisors to rate the 
quality of school leaders' practices. The department piloted the tool in school 
years 201 2/1 3 and 201 3/1 4 and requested an independent study of the pilot 
data to assess how well this tool measures the effectiveness of school leaders. 
This interim report is based on data from 2012/13. 

2. What is the Framework for Leadership? 

The Framework for Leadership (FFL) was developed by the Pennsylvania 
Department of Education to measure the quality of school leaders' 
practices. The FFL specifies 1 9 leadership practices on which each school 
leader is rated by an administrator — such as a superintendent or assistant 
superintendent — who supervises that school leader. On each practice, a 
school leader can receive a rating of distinguished (3 points), proficient 
(2 points), needs improvement (1 point), or failing (0 points). 


The 19 practices are grouped into four categories: (1) strategic/cultural 
leadership, (2) systems leadership, (3) leadership for learning, and 
(4) professional and community leadership. School leaders also receive 
an overall rating for each category. The ratings that supervisors assign 
are supposed to be based on evidence submitted by the school leaders 
and the supervisors'direct observation of the leaders' practices. 

3. Why was it important to gather evidence on the 
accuracy of the Framework for Leadership? 

Analyzing data from the pilot of the Framework for Leadership can help 
education officials in Pennsylvania refine and improve this evaluation tool 
before it is used in formal evaluations of school leaders. 

KEY FINDINGS AND LESSONS 
FROM THE STUDY 


4. What strength of the Framework for Leadership did 
this study find? 

The strength of the Framework for Leadership (FFL) is that different parts of 
the tool were consistent with each other in producing conclusions about a 
school leader's effectiveness. School leaders who earned higher scores in one 
category of leadership practices also tended to earn higher scores in the other 
categories. This finding suggests that different parts of the FFL are connected 
by a coherent definition of leadership quality. 
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5. What shortcomings of the Framework for Leadership 
did this study find? 

This study identified two shortcomings of the Framework for Leadership (FFL): 

i. FFL scores did not differ much among school leaders. Most ratings were 
high. On specific leadership practices, supervisors rated principals and 
assistant principals as proficient or distinguished 95 percent of the time and 
as needing improvement in the remaining 5 percent. Failing ratings were 
extremely rare. In contrast, prior research has shown that principals differ 
considerably in their effectiveness at raising student achievement. 

ii. FFL scores were not associated with estimates of school leaders' 
contributions to student achievement growth. In this pilot phase, the 
FFL did not successfully distinguish between school leaders who were 
more effective in raising student achievement and those who were less 
effective in doing so. School leaders with larger estimated contributions 
to student achievement growth did not, on average, receive higher FFL 
scores than school leaders with smaller estimated contributions to student 
achievement growth. 

6. Is the Framework for Leadership any more or less 
accurate than other tools for rating school leaders? 

It is unknown whether other tools are more or less accurate than the 
Framework for Leadership. For nearly all school leader evaluation tools, there 
have been no studies documenting how well those tools indicate school 
leaders' contributions to student achievement growth. The few tools that 
have been examined have not been shown to be consistently associated with 
principals' contributions to student achievement growth. Moreover, there is 
no prior analysis of how accurately school leader evaluation tools measure 
assistant principals' contributions. 

7. What lessons can be drawn from the study findings 
to improve the Framework for Leadership? 

Since the range of principals' evaluation scores was so limited, the Pennsylvania 
Department of Education may need to provide more specific guidance on how 
to determine ratings. For example, the department could provide concrete 
examples of the quantity and quality of evidence that would merit each possible 
score for every specific leadership practice. These examples could set a higher 
standard for scoring well than the standards that supervisors used in 201 2/1 3. 

In addition, more evidence is needed on how well the Framework for 
Leadership (FFL) can identify effective and ineffective school leaders. This 
study found that FFL scores were not related to school leaders' contributions to 
student achievement growth in 2012/13; however, achievement growth does 
not have to be the only point of comparison. The department could obtain 
additional measures of school leaders' performance, such as anonymous 
ratings by teachers, to determine whether these other measures can 
corroborate school leaders' FFL scores. 
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9. Who sponsored this study? 

This study was sponsored by the Regional Educational Laboratory Mid-Atlantic, 
an organization established by the U.S. Department of Education to help states 
and districts in the Mid-Atlantic Region use data and research to improve 
students'academic outcomes. 

10. Who conducted this study? 

Mathematica Policy Research conducted this study as a subcontractor to 
ICF International, which operates the 201 2-201 7 Regional Educational 
Laboratory Mid-Atlantic. 

1 1 . What was the Pennsylvania Department of 
Education's role in this study? 

The Pennsylvania Department of Education requested this study to help it 
refine and enhance the Framework for Leadership (FFL). The department also 
provided the data (FFL scores and other administrative data on students and 
school leaders) used in this study. The department did not fund the study, 
conduct the analysis, or contribute to the report. 

1 2. What were the limitations of this study? 

This study had three primary limitations: 

i. All measures of school leaders' contributions to achievement growth were 
based on student outcome measures from state tests. Due to the lack of 
data, this study did not examine whether Framework for Leadership (FFL) 
scores reflect school leaders' contributions to nontest outcomes (such as 
students' creativity and character). 

ii. Although this study developed a new analytic method for measuring 
school leaders' contributions to student achievement growth (as a point 
of comparison for FFL scores), there is not yet a clear consensus among 
researchers on the best approach to measuring these contributions. 

iii. The sample size of the 201 2/1 3 pilot was not large enough to produce 
very precise estimates of the relationships between FFL scores and school 
leaders' contributions to student achievement growth. Findings may 
change in the next pilot phase, when more data will be available. 


STUDY DESIGN, PARTICIPANTS, 

AND STAKEHOLDERS 

8. Who participated in this study? 

A total of 336 principals and 69 assistant principals were evaluated in the 
201 2/1 3 pilot study. These school leaders came from 344 schools spread across 
146 school districts and other local education agencies. Most school districts 
in the study were required to participate because they received funds from the 
federal Race to the Top program. In addition, schools receiving federal School 
Improvement Grants to implement a transformation model of improvement 
were required to participate, and some school districts voluntarily selected 
schools to participate. 


FUTURE STEPS 


1 3. What comes next? 

Data from the 2013/14 pilot year of the Framework for Leadership (FFL) will 
be analyzed in a similar fashion to determine how well FFL scores measure 
school leaders' effectiveness. The Pennsylvania Department of Education 
has projected more participants in 2013/14, which will permit more precise 
analyses. In addition, data will be available on how persistently students stay 
enrolled in high schools, so further analyses will assess how closely FFL scores 
are associated with high school leaders' contributions to students' enrollment 
persistence. A final report on the findings from school years 201 2/1 3 and 
2013/14 is scheduled to be released in 2015. 


